Variance-based Regularization with Convex Objectives

We develop an approach to risk minimization and stochastic optimization that pro-1vides a convex surrogate for variance, allowing near-optimal and computationally2efficient trading between approximation and estimation error. Our approach builds3off of techniques for distributionally robust optimization and Owen’s empirical4likelihood, and we provide a number of finite-sample and asymptotic results char-5acterizing the theoretical performance of the estimator. In particular, we show that6our procedure comes with certificates of optimality, achieving (in some scenarios)7faster rates of convergence than empirical risk minimization by virtue of auto-8matically balancing bias and variance. We give corroborating empirical evidence9showing that in practice, the estimator indeed trades between variance and absolute10performance on a training sample, improving out-of-sample (test) performance11over standard empirical risk minimization for a number of classification problems.12